2025-06-13 15:45:20,222 [ 563927 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2025-06-13 15:45:20,222 [ 563927 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2025-06-13 15:45:20,222 [ 563927 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2025-06-13 15:45:20,222 [ 563927 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_5j4vzl --privileged --dns-search='.' --memory=30709035008 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=1 --color=no --durations=0 --report-log=parallel2_1.jsonl --report-log-exclude-logs-on-passed-tests test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_update_one_cluster -vvv " altinityinfra/integration-tests-runner:ad96270260ff '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [2 items] scheduling tests via LoadFileScheduling test_reload_clusters_config/test.py::test_delete_cluster [gw0] [ 50%] PASSED test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_update_one_cluster [gw0] [100%] FAILED test_reload_clusters_config/test.py::test_update_one_cluster =================================== FAILURES =================================== ___________________________ test_update_one_cluster ____________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_update_one_cluster(started_cluster): send_repeated_query("distributed") send_repeated_query("distributed2") > assert get_errors_count("test_cluster") > 0 E AssertionError: assert 0 > 0 E + where 0 = get_errors_count('test_cluster') test_reload_clusters_config/test.py:202: AssertionError ------------------------------ Captured log call ------------------------------- 2025-06-13 15:48:10 [ 700 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-06-13 15:48:22 [ 700 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-06-13 15:48:35 [ 700 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-06-13 15:48:47 [ 700 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-06-13 15:48:59 [ 700 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-06-13 15:49:13 [ 700 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-06-13 15:49:25 [ 700 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-06-13 15:49:37 [ 700 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-06-13 15:49:49 [ 700 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-06-13 15:50:02 [ 700 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-06-13 15:50:14 [ 700 ] DEBUG : Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node (cluster.py:3570, query) ---------------------------- Captured log teardown ----------------------------- 2025-06-13 15:50:15 [ 700 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/.env --project-name roottestreloadclustersconfig-gw0 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] (cluster.py:121, run_and_check) 2025-06-13 15:50:22 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Stopping (cluster.py:147, run_and_check) 2025-06-13 15:50:22 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Stopped (cluster.py:147, run_and_check) 2025-06-13 15:50:22 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-06-13 15:50:22 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-06-13 15:50:22 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-06-13 15:50:22 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-06-13 15:50:22 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-06-13 15:50:22 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-06-13 15:50:22 [ 700 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2025-06-13 15:50:22 [ 700 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/.env --project-name roottestreloadclustersconfig-gw0 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] (cluster.py:121, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Stopping (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Stopped (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Removing (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Removed (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Removing (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Removing (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Removing (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Removed (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Removed (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Removed (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Network roottestreloadclustersconfig-gw0_default Removing (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stderr: Network roottestreloadclustersconfig-gw0_default Removed (cluster.py:147, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Cleanup called (cluster.py:846, cleanup) 2025-06-13 15:50:23 [ 700 ] DEBUG : Docker networks for project roottestreloadclustersconfig-gw0 are NETWORK ID NAME DRIVER SCOPE (cluster.py:825, print_all_docker_pieces) 2025-06-13 15:50:23 [ 700 ] DEBUG : Docker containers for project roottestreloadclustersconfig-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:833, print_all_docker_pieces) 2025-06-13 15:50:23 [ 700 ] DEBUG : Docker volumes for project roottestreloadclustersconfig-gw0 are DRIVER VOLUME NAME (cluster.py:841, print_all_docker_pieces) 2025-06-13 15:50:23 [ 700 ] DEBUG : Command:[docker container list --all --filter name='^/roottestreloadclustersconfig-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Unstopped containers: {} (cluster.py:860, cleanup) 2025-06-13 15:50:23 [ 700 ] DEBUG : No running containers for project: roottestreloadclustersconfig-gw0 (cluster.py:874, cleanup) 2025-06-13 15:50:23 [ 700 ] DEBUG : Trying to prune unused networks... (cluster.py:880, cleanup) 2025-06-13 15:50:23 [ 700 ] DEBUG : Trying to prune unused images... (cluster.py:896, cleanup) 2025-06-13 15:50:23 [ 700 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Images pruned (cluster.py:899, cleanup) 2025-06-13 15:50:23 [ 700 ] DEBUG : Trying to prune unused volumes... (cluster.py:905, cleanup) 2025-06-13 15:50:23 [ 700 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2025-06-13 15:50:23 [ 700 ] DEBUG : Volumes pruned: 1 (cluster.py:910, cleanup) ----------------- generated report log file: parallel2_1.jsonl ----------------- ============================== slowest durations =============================== 138.46s call test_reload_clusters_config/test.py::test_delete_cluster 124.52s call test_reload_clusters_config/test.py::test_update_one_cluster 26.11s setup test_reload_clusters_config/test.py::test_delete_cluster 8.13s teardown test_reload_clusters_config/test.py::test_update_one_cluster 0.00s teardown test_reload_clusters_config/test.py::test_delete_cluster 0.00s setup test_reload_clusters_config/test.py::test_update_one_cluster =========================== short test summary info ============================ FAILED test_reload_clusters_config/test.py::test_update_one_cluster - Asserti... PASSED test_reload_clusters_config/test.py::test_delete_cluster =================== 1 failed, 1 passed in 299.08s (0:04:59) ==================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 492, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_5j4vzl --privileged --dns-search='.' --memory=30709035008 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=1 --color=no --durations=0 --report-log=parallel2_1.jsonl --report-log-exclude-logs-on-passed-tests test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_update_one_cluster -vvv " altinityinfra/integration-tests-runner:ad96270260ff ' returned non-zero exit status 1.